SERVER-130128: fix(sharding): enqueue delete for deferred updates moving out of migrating chunk#1819
Conversation
…ating chunk If a deferred transaction update moves a document out of a migrating chunk and the document is subsequently deleted before `_processDeferredXferMods` runs, the destination shard misses the deletion, leaving an orphaned document. This commit updates `_processDeferredXferMods` to model a delete if the document is no longer found and its pre-image was in the chunk.
Abuhaithem
changed the title
what happend to my issue in jira @kelly-cs ? |
|
Hey @Abuhaithem you should be able to see this one again. Our teams are taking a look still |
|
I don't think this is an issue today. This is because for the issue to happen, it needs the following condition:
There are 2 things that saves this from happening:
I still think that the scenario is interesting, so I'll update our tests. |
Agreed @renctan ! with SERVER-71028 blocking the cloner and current-term ops always carrying post-image doc keys, I don't see a live path to the orphan either. But the correctness here depends on those two non-local invariants, and the existing comment states a guarantee ('the delete would have been captured') that isn't actually true ; the delete is range-filtered and only covered via the update's modeled delete. I'd suggest keeping the range-check-and-delete as cheap defense-in-depth (it's a no-op on the recipient when the doc was never transferred), or at minimum correcting the comment to document the real invariant. Happy to help with the test updates : ) |
3 participants
Fixes: SERVER-130128
This pull request addresses a severe data inconsistency bug where an orphaned document can be left on a migration destination shard.
When a chunk migration is ongoing, the migration cloner tracks updates to documents inside the chunk's range. For transactions prepared on the donor in a previous term but committed in the current term, the oplog entry lacks a
postImagedocument key. The cloner handles these by deferring their processing via_deferProcessingForXferModand resolving them later duringnextModsBatchvia_processDeferredXferMods.However, a serious race condition exists if the deferred update moves a document out of the migrating chunk, and the document is subsequently deleted before
_processDeferredXferModscan run:postImageDocKeyis empty, and the update is deferred.onDeleteOpevaluates the deletion. Because the document's new shard key is out of bounds for the migrating chunk,onDeleteOpreturns early and does not enqueue a deletion toxferMods._processDeferredXferModsruns. It attempts to fetch the document usingHelpers::findById.findByIdfails. The code previously assumed: "That delete would have been captured by the xferMods so nothing else to do here," and executedcontinue.Because the deletion was skipped by
onDeleteOp(as it occurred outside the chunk bounds), the destination shard is never instructed to delete the document. The migration completes with the document incorrectly cloned to the destination shard, resulting in an orphaned document.The Fix:
This PR fixes the bug by having
_processDeferredXferModsextract the shard key from the deferred update'spreImageDocKey. If thepreImageDocKeywas within the bounds of the migrating chunk, a deletion is unconditionally enqueued for the recipient shard. This operation is fully idempotent and ensures the destination shard correctly deletes the document.Impact & Severity
High Severity: Silently compromises data integrity for sharded clusters during migrations combined with failovers and prepared transactions. These orphaned documents could reappear in queries or violate unique constraints.
Type of Change
Testing